OpenVLHarness

Real World Visual Agentic Tool-Use with Multimodal Memory

Yu Zhou1*, Cheng-Fu Yang1*, Rui Sun1*, Di Wu1*, Sibo Peng1, Mingyang Zhang1, Zhen Yang2, Ruiyi Zhang2, Nanyun Peng1, Zhe Gan2, Kai-Wei Chang1

1University of California, Los Angeles  ·  2Apple
*Equal contribution

Paper Code Data Demo

Highlights

Gains Across Model Families Average over 22 benchmarks · Base vs. + OpenVLHarness
Base MLLM + OpenVLHarness
GPT-6 Performance vs. Cost 22 benchmarks · labels: reasoning effort

Architecture

Recorded trajectories, replayed step by step. Select an example, pause or resume playback, step through it, or click a panel to expand it.

Domain
User input
Answer: …
MLLM orchestrator
idle
Environment Update
Tool call
Processed tool outputs
Capability Layer
Perception
Zoom-in
Grounding
OCR
Depth
Camera pose
Search
Text search
Image search
Webpage visit
Coding
Coding Agent
Models & services
Segment Anything Depth Anything PaddleOCR‑VL Search engines Coding modelorchestrator MLLM Image processors

Demo

The agent loop and perception tools run on our server; your model (OpenAI, or an OpenAI-compatible endpoint with tool calling) orchestrates, writes the coding tool's Python and summarizes web results.

Connecting to the demo server…
Samples web = requires a Serper key
Orchestrator model

Keys are used for the request only and are not stored.
A Serper key is free: 2,500 queries, no credit card.

Trajectory
  1. The trajectory appears here while the agent runs.

Per-Benchmark Results

Bold = best within each backbone group; underline = best across all shown backbones. Qwen3-VL results are means of three runs; GPT-6 and Kimi K3 results are single runs. GPT-6 models use medium reasoning effort and Kimi K3 uses max. GPT-6 Sol and Kimi K3 are from the appendix (Table 7). OdinW-13-All asks for all categories of an image's source dataset in one query.

Analysis

Per-Benchmark Gains

Base and OpenVLHarness scores on each of the 22 benchmarks, per backbone. Average improvement over the base model ranges from +9.7 to +12.6 points across six backbones; with GPT-6 Astra, all 22 benchmarks improve.

Base + OpenVLHarness

Comparison with Other Harnesses

Average over 22 benchmarks with the same backbone. GPT-6 Luna: Base 45.8, Visual Sketchpad 45.6, Codex 43.8, OpenVLHarness 56.5. Qwen2.5-VL-7B: AdaReasoner 25.3, PixelReasoner 28.0, DeepEyesV2 25.0 (fine-tuned for tool use); OpenVLHarness 33.5 (original weights).

Component-Wise Ablation

Components added cumulatively to the no-tool baseline, Qwen3-VL-8B and 32B. Each cell shows the score and its change from the row above.

Shading: change from the previous configuration ( gain, loss). Source: paper Table 3.

Tool Usage by Domain

Share of tool calls in each domain, per backbone, over all evaluated questions. Micro counts every call; per-instance counts each tool at most once per question. Totals are given below each panel.

Source: paper Fig. 7 (appendix) telemetry. Micro view reproduces the figure.

Harness Optimization

Kimi K3 critiques 9,893 failed Qwen3-VL-8B trajectories and assigns each a root cause and a remedy channel: new task-specific tools (A–C), harness text (Y), or none (Z). The dominant channel selects the workflow. Harness search edits the system prompt, tool descriptions, and output templates; tool generation composes existing capabilities into new tools. Both are run by a coding agent with model weights fixed.

Failure Causes and Remedy Channels by Domain Qwen3-VL-8B · 9,893 failures · unweighted mean over datasets per domain
Remedy Profiles and Specialization Gains Qwen3-VL-8B · gains over the unoptimized harness
New task-specific tools (A+B+C) Harness text (Y) No remedy (Z)

Citation

@article{openvlharness2026,
  title   = {OpenVLHarness: Real World Visual Agentic Tool-Use with Multimodal Memory},
  author  = {Zhou, Yu and Yang, Cheng-Fu and Sun, Rui and Wu, Di and Peng, Sibo and Zhang, Mingyang and Yang, Zhen and Zhang, Ruiyi and Peng, Nanyun and Gan, Zhe and Chang, Kai-Wei},
  journal = {arXiv preprint arXiv:2610.26860},
  year    = {2026},
  url     = {https://arxiv.org/abs/2610.26860}
}